100 research outputs found
Recommended from our members
Shining Light in Dark Places: A Study of Anonymous Network Usage ; CU-CS-1032-07
Global-Liar: Factuality of LLMs over Time and Geographic Regions
The increasing reliance on AI-driven solutions, particularly Large Language
Models (LLMs) like the GPT series, for information retrieval highlights the
critical need for their factuality and fairness, especially amidst the rampant
spread of misinformation and disinformation online. Our study evaluates the
factual accuracy, stability, and biases in widely adopted GPT models, including
GPT-3.5 and GPT-4, contributing to reliability and integrity of AI-mediated
information dissemination.
We introduce 'Global-Liar,' a dataset uniquely balanced in terms of
geographic and temporal representation, facilitating a more nuanced evaluation
of LLM biases. Our analysis reveals that newer iterations of GPT models do not
always equate to improved performance. Notably, the GPT-4 version from March
demonstrates higher factual accuracy than its subsequent June release.
Furthermore, a concerning bias is observed, privileging statements from the
Global North over the Global South, thus potentially exacerbating existing
informational inequities. Regions such as Africa and the Middle East are at a
disadvantage, with much lower factual accuracy. The performance fluctuations
over time suggest that model updates may not consistently benefit all regions
equally.
Our study also offers insights into the impact of various LLM configuration
settings, such as binary decision forcing, model re-runs and temperature, on
model's factuality. Models constrained to binary (true/false) choices exhibit
reduced factuality compared to those allowing an 'unclear' option. Single
inference at a low temperature setting matches the reliability of majority
voting across various configurations. The insights gained highlight the need
for culturally diverse and geographically inclusive model training and
evaluation. This approach is key to achieving global equity in technology,
distributing AI benefits fairly worldwide.Comment: 24 pages, 12 figures, 9 table
FraudDroid: Automated Ad Fraud Detection for Android Apps
Although mobile ad frauds have been widespread, state-of-the-art approaches
in the literature have mainly focused on detecting the so-called static
placement frauds, where only a single UI state is involved and can be
identified based on static information such as the size or location of ad
views. Other types of fraud exist that involve multiple UI states and are
performed dynamically while users interact with the app. Such dynamic
interaction frauds, although now widely spread in apps, have not yet been
explored nor addressed in the literature. In this work, we investigate a wide
range of mobile ad frauds to provide a comprehensive taxonomy to the research
community. We then propose, FraudDroid, a novel hybrid approach to detect ad
frauds in mobile Android apps. FraudDroid analyses apps dynamically to build UI
state transition graphs and collects their associated runtime network traffics,
which are then leveraged to check against a set of heuristic-based rules for
identifying ad fraudulent behaviours. We show empirically that FraudDroid
detects ad frauds with a high precision (93%) and recall (92%). Experimental
results further show that FraudDroid is capable of detecting ad frauds across
the spectrum of fraud types. By analysing 12,000 ad-supported Android apps,
FraudDroid identified 335 cases of fraud associated with 20 ad networks that
are further confirmed to be true positive results and are shared with our
fellow researchers to promote advanced ad fraud detectionComment: 12 pages, 10 figure
- …